Predicting Visual Intelligibility Gain in the SynFace Application

When the acoustic signal is weak or disturbed, normal hearing people as well as hearing impaired make use of visual information as a support for their interpretation of the speech. Thus, a talking head could be of great support for the hearing impaired when engaged in speech activities where no visual information is available, such as a telephone conversation or when listening to an audio book. SynFace, a talking head developed at the Royal Institute of Technology, is an application where the lip movements are driven by the acoustic signal and synchronized with the speech. Thus, it could be a valuable support since it provides sufficient visual information. The purpose of this project is to find an error metric that could be used to predict the visual intelligibility gain when using the SynFace application. The metric is based on visual similarity among Swedish vowels and consonants, respectively. The perceptual distances were established in an experiment where normal hearing subjects watched silent movie clips where nonsense words were presented by a synthetic as well as a natural face. Error metrics were calculated for three different networks using frame-by-frame comparison, where the perceptual distances were used as weights for the phoneme recognition errors. The calculated error metrics were then mapped to intelligibility in regard to SRT (Speech Reception Threshold) reduction in noise. A linear relationship between the performance of the network and the SRT levels was found. Visuell uppfattbarhetsprediktion i SynFace